Papers with preference tuning
CiPO: Counterfactual Unlearning for Large Reasoning Models through Iterative Preference Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to unlearning large reasoning models do not remove unwanted knowledge from CoT traces or interfere with the reasoning process. |
| Approach: | They propose a framework that targets the CoT reasoning in Large Reasoning Models by generating a valid counterfactual reasoning trace for preference tuning. |
| Outcome: | Experiments on large LRMs show that CiPO completely removes knowledge from the intermediate CoT steps and the final answer while preserving the reasoning abilities of LRM. |
Constructing Your Model’s Value Distinction: Towards LLM Alignment with Anchor Words Tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study of large language models (LLMs) shows that they can generate outputs that are honest, positive, harmless, etc. |
| Approach: | They propose a method that amplifies logits difference between positive and negative tokens . they propose to use the logits gap to generate positive and positive tokens after alignment . |
| Outcome: | The proposed method achieves effective alignment, but requires fewer computational resources compared to training-time alignment methods. |
ComPO: Community Preferences for Language Model Personalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for training language models with human feedback rely on subjective preferences that are assumed to account for an "average" user . however, annotating preferences is inherently subjective and results in generic models that generate outputs not preferred by many user groups. |
| Approach: | They propose a method to personalize preference optimization in LMs by contextualizing the probability distribution of model outputs with the preference provider. |
| Outcome: | The proposed method improves performance by focusing on group-level preferences rather than individual feedback. |
Improving Alignment in LVLMs with Debiased Self-Judgment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for aligning LVLMs rely on external datasets, human annotations or complex post-processing. |
| Approach: | They propose a method that generates a debiased self-judgment score for LVLMs . this self-evaluation metric is created internally by the model without external resources . |
| Outcome: | The proposed approach outperforms existing methods in reducing hallucinations and safety concerns. |
ReMedi: Reasoner for Medical Clinical Prediction (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to predicting future clinical outcomes from EHRs focus on enhancing medical knowledge through distillation or RAG while relying on the model’s internal ability to interpret contextual information. |
| Approach: | They propose a framework for improving clinical outcome prediction from EHR using a sample regeneration mechanism that leverages ground-truth answers as hints to enhance reasoning. |
| Outcome: | Experiments on multiple EHR prediction tasks show significant gains of up to 19.9% over state-of-the-art baselines in terms of F1 score, underscoring ReMedi’s effectiveness in real-world clinical prediction. |
T2A-Feedback: Improving Basic Capabilities of Text-to-Audio Generation via Fine-grained AI Feedback (2025.acl-long)
Copied to clipboard
Zehan Wang, Ke Lei, Chen Zhu, Jiawei Huang, Sashuai Zhou, Luping Liu, Xize Cheng, Shengpeng Ji, Zhenhui Ye, Tao Jin, Zhou Zhao
| Challenge: | Text-to-audio (T2A) models still struggle to satisfy human preferences for prompt-following and acoustic quality when generating complex multi-event audio. |
| Approach: | They propose to use AI feedback learning to enhance basic capabilities of text-to-audio models . they use a large audio preference dataset to evaluate the model's capabilities . |
| Outcome: | The proposed model improves in simple and complex scenarios with AI feedback learning. |
Reranking-based Generation for Unbiased Perspective Summarization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation frameworks rely on traditional metrics for measuring key attributes such as coverage and faithfulness without verifying their applicability. |
| Approach: | They propose to use human annotations to measure perspective summary quality and reranking-based methods yield strong results. |
| Outcome: | The proposed methods show that they perform well with synthetically generated and reranking-labeled data. |
CARE: Multilingual Human Preference Learning for Cultural Awareness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language Models are tuned with human preferences to produce helpful responses, but the impact of preference tuning on the ability to handle culturally diverse queries remains understudied. |
| Approach: | They introduce a multilingual resource that contains culturally specific questions and 31.7k responses with human judgments. |
| Outcome: | The proposed model outperforms models with stronger initial cultural performance . the proposed model has gaps in the literature on culturally relevant data . |